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Abstract Body 


Background / Context: 

In recent years, there has been an increased interest in assessing the effects of educational 
interventions via experimental designs where students, classrooms, or schools are randomly 
assigned to a treatment and a control condition. An important part of the design phase of an 
experiment involves power analysis. Statistical power is the probability of detecting the 
treatment effect of interest when it exists (Boruch & Gomez, 1977; Cohen, 1988). A priori power 
computations are critical in designing experiments because they infonn empirical researchers 
about the sampling scheme needed to detect a treatment effect. Specifically, a priori power 
analyses help educational researchers identify how big a sample is needed at the student, 
classroom, or school level to ensure a high probability (e.g. , > 80 percent) of detecting a 
treatment effect if it were true (Lipsey 1990). 

Education experiments incorporate often times a longitudinal component where students are 
followed over time and are measured several times. In repeated measures experiments each 
student has their own trajectory which is a function of time and indicates the rate of change over 
time (Raudenbush & Bryk, 2002). The change in measurements over time does not always 
follow a linear trend. Instead, trajectories sometimes point to nonlinearities such as curvilinear 
trends. One way of defining trajectories of change is via polynomial functions (Raudenbush & 
Liu, 2001). The first degree polynomial indicates linear rate of change, the second degree 
polynomial indicates a quadratic rate of change, the third degree polynomial indicates a cubic 
rate of change and so forth. 

Studies about polynomial change may be viewed as having a nested structure. For example, 
measurements are nested within individuals and this nesting needs to be taken into account in the 
design phase of the study as well as in the statistical analysis phase. Raudenbush and Liu (2001) 
provided power analysis and sample determination methods for repeated measures in two-level 
models, where repeated measures are nested in individuals. They showed that power is a function 
of the magnitude of the treatment effect, the sample size of individuals, the study duration, and 
the frequency of measurements over time. Researchers should take into account all of these 
parameters in the design phase of the experiment to ensure that treatment effects will be detected. 

Populations in education have frequently more complicated structures. For example, 
students are also nested within classes or schools and so forth. In addition, education 
interventions typically assign either schools or students randomly to treatment or control groups. 
For instance, students are assigned to small or regular classes within schools. Or schools are 
randomly assigned to an assessment program or not. It seems natural to extend methods for 
power analysis for tests of treatment effects in studies of polynomial change from two to three- 
levels. Spybrook et al. (2011) reported in the optimal deign manual formulae to calculate power 
for three-level polynomial change models without covariates, where treatment is at the third 
level. 

Purpose / Objective / Research Question / Focus of Study: 

The purpose of this study is extend previous methods by Raudenbush and Liu (2001) and 
Spybrook et al. (2011), and provide methods for power analysis of tests of treatment effects in 
studies of polynomial change with two levels of nesting (e.g., students and schools) where the 
treatment is either at the third level (e.g., school intervention) or at the second level (e.g., student 
intervention). In particular, we provide methods for power analysis for block randomized designs 
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(BRD) where the treatment is at the second level (e.g., student intervention) and the third level 
units (e.g., schools) serve as blocks. Please note that methods for power analysis for cluster 
randomized designs (CRD), where for instance schools are randomly assigned in a treatment and 
a control group, is omitted in this proposal because of words limits. 

Significance / Novelty of study: 

This study fills the gap in literature through providing methods for power analysis of 
detecting treatment effects in studies of polynomial change with two levels of nesting. 


Statistical Model: Treatment Assigned at Second Level (Block Design) 

Consider a simple three-level growth design where level-2 units (e.g., students) are 
randomly assigned to treatment or control conditions. The first level for change over time of 
level-2 unit i in cluster j can de expressed as a polynomial function, namely 


Ygij °~0ij C 0g + °\i] C \g + a 2j C 2g + ••• + a (P-\)ij C (P-\)g + U gij 


( 1 ) 


where c represent orthogonal polynomial contrasts of degree p (p = 0, 1, ..., P -\ ) at 

measurement g (g = 1, ..., G), a .. ’s represent the mean and the rates of change (linear, 

quadratic, cubic, etc.), and u gij is the within level-2 unit random term with variance cr . We 

work with orthogonal polynomial contrasts because they facilitate the computations of estimators 
and their standard errors, and simplify power analysis (see Raudenbush & Liu, 2001). The results 
apply to studies of any length and for polynomials of any degree (Kirk, 2013). 

The level-2 model incorporates the treatment ( 7)/), namely 


«rt,=/L/ + /W) + C. 


( 2 ) 


where J3 p0j ’s represent the average polynomial effects within level-3 units, 73 is a dummy 
variable coded as one if second level unit i in third level unit j is assigned to treatment condition 
and zero otherwise, /3 pXj is the treatment effect within level-3 units, and the % pij ’s are level-2 
random effects within level-3 units for each polynomial change parameter. The random effects 
follow a multivariate normal distribution with zero means, variances r pp , and covariance r pp , , 
between random effects % pij and % pHj . 

The third level equations for the intercept ( J3 p0j ) and the treatment effect ( J3 plj ) are 

PpOj = Yp00 +r lp0j 

P„M=r t io+'Vj' 


where y p00 ’s represent the average polynomial effects across level-3 units, the rj p0j ’s are level-3 
unit specific random effects for each polynomial change parameter, y pW ’s represent the average 
difference between the treatment and the control groups for each polynomial change parameter 
across level-3 units, and the tj Xj ' s are treatment by level-3 unit random effects (interaction 

effects) for each polynomial change parameter. The r/ p0j ’s follow a multivariate normal 
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distribution with zero means and variances co 2 p0 , whilst the treatment by level-3 unit random 

effects also follow a normal distribution with a mean of zero and a variance col , where the 

subscript T indicates treatment at the second level whose effect varies at the third level. 

Suppose there are M level-3 units and n level-2 units within treatment or control condition 
within each level-3 unit, which means that the total number of level-2 units in each level-3 unit is 
N=2n. Then, the estimate of the variance of the treatment effect for polynomial p is 


Var (r P w ) = nc *rpp + + °l) ’ / Z 4 - 


( 4 ) 


*=i 


Suppose that a researcher wants to test the hypothesis that y l0 is different from zero and 
carries out a /-test. The test statistic is defined as 


t = Y P 10 / yjVar(y p 10 ). 


( 5 ) 


When the null hypothesis is true, the test statistic / has a Student’s /-distribution with M-l 
degrees of freedom (Konstantopoulos, 2008). When the null hypothesis is false, the test statistic 
/ has the non-central /-distribution with M-l degrees of freedom and non-centrality parameter 1 . 
The non-centrality parameter is defined as the expected value of the estimate of the treatment 
effect divided by the square root of the variance of the estimate of the treatment effect, namely 



1 
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We define the standardized effect size for a polynomial degree p as 


ES = 


y pw 
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Then, the non-centrality parameter l of the /-test simplifies to 


( 6 ) 

( 7 ) 
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The power of a two-tailed /-test for a specified significance level a is defined as 
pi = 1 - H [c(<x /2, M-l), {M-l), 2] + H [-c( a /2, M-l), {M-l), 2] 


( 8 ) 

( 9 ) 


where c{a,v) is the level a one-tailed critical value of the /-distribution with v degrees of freedom, 
and H(x, v, 2) is the cumulative distribution function of the non-central /-distribution with v 
degrees of freedom and non-centrality parameter 2. 

When covariates are included at the second level, the non-centrality parameter l a becomes 



2 2 

CO T + T 
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where subscript^ indicates adjustment because of covariates and 


w , 


®RTpp ! ®Tpp 1 W 2 


x lx 1 

L Rpp ' L pp : 


( 10 ) 


( 11 ) 


where the subscript R indicates residual variance because of covariates. That is, wi indicates the 
proportion of the variance at the second level that is still unexplained, and W 3 indicates the 
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proportion of the treatment by level-3 unit variance at the third level that is still unexplained. 
Then, the power of a two-tailed /-test for a specified significance level a is defined as 

P 2 = 1 - H [c( a /2, M-q-1 ), (M-q-1), h] + H [-c( a /2, M-q-1 ), (M-g-7), U,], (12) 

where q is the number of covariates at the third level. 

Usefulness / Applicability of Method: 

To illustrate the applicability of the methods to assess consequences of the study duration, 
sample sizes (students and schools), and covariates on power, we utilized data from Project 
STAR (Student-Teacher Achievement Ratio) in Tennessee (e.g., Finn & Achilles, 1990). This 
experiment employed a block randomized design, where within each school (the block) and 
grade, students and their teachers were randomly assigned to small classes and regular-size 
classes. 

We fitted a three level linear change model, where treatment is at the second level, and got 
variance estimates 

of = 0.060738, rjj =0.00753, co 2 Tn = 0.02097 

Using these variance estimates, we explore how powers are influenced by the study duration, 
sample size, and covariates. Figure 1-3 show how variations of the study duration and sample 
sizes affect the power to detect the linear rate of change of the treatment effect of interest in 
block designs, assuming two-tailed /-tests at the 0.05 significance level and effect size as 0.40. 
The power of detecting a linear rate of change does not increase consistently as study duration 
increases. Similarly, power does not increase consistently as the number of students increases, 
especially after a certain number of students. However, as the number of schools increases, 
power increases significantly more. Such results indicate that to boost power it is recommended 
to sample more schools rather than to sample more students per school. 

(please insert figure 1 here) 

(please insert figure 2 here) 

(please insert figure 3 here) 

Figure 4 shows how the power is influenced by the proportion of unexplained variance at the 
second and third levels. The results show that power increases when covariates are added in the 
model, as expected. 

(please insert figure 4 here) 


Conclusions: 

The power of the test of the treatment effect in studies of polynomial change with two-levels 
of nesting is a function of the magnitude of the treatment effect, the study duration, the sample 
size of individuals, the sample size of clusters, and the proportion of the variances that covariates 
at the second or third levels explain. 

Several findings emerged from this study that applies to both CRD and BRD. First, power 
increases as the study duration, the number of students in each school, or the number of schools 
increases. Other things being equal, the number of level-3 units (clusters) influences power more 
than the number of level-2 units (individuals) or the duration of the study. This indicates clearly 
that researchers should sample more schools rather than students within schools to maximize 
power. Second, covariates that explain a proportion of variances at the second or third level 
could increases powers and thus reduce the study duration or sample sizes needed to boost power 
to a certain level. 
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Appendix B. Figures 



-• M = 20 ♦ M = 40 

M = 60 ■ M = 80 


Figure 1. Effect of Study Duration ( D ) and Number of Schools (M) on Power Holding Number of 
Students ( N) in Each School Constant at 40: BRD, Linear Rate of Change 
Note: Effect size is 0.4 with a significance level of 0.05. 


B-l 



Figure 2. Effect of Study Duration ( D ) and Number of Students (AO on Power Flolding Number of 
Schools ( M) Constant at 40: BRD, Linear Rate of Change 
Note: Effect size is 0.4 with a significance level of 0.05. 
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-• N = 40 ♦ N = 80 

■* N = 120 • N = 160 


Figure 3. Effects of Number of Schools (M) and Number of Students (AO on Power Holding Study 
Duration ( D ) Constant at 4: BRD, Linear Rate of Change 
Note: Effect size is 0.4 with a significance level of 0.05. 
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W2 = 0.1 ♦ W2 = 0.3 

■A W2 = 0.5 • W2 = 0.7 


Figure 4. Effect of Covariates on Power: BRD, Linear Rate of Change 

Note: The study duration is 4 with 40 schools and 40 students in each school; significance level is 0.05. 
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